fix(voice): include conversation items added during tool execution in reply context#6084
fix(voice): include conversation items added during tool execution in reply context#6084ShayneP wants to merge 1 commit into
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| # Conversational analog of the update_instructions() refresh below. | ||
| known_ids = {item.id for item in chat_ctx.items} | ||
| known_ids.update(item.id for item in tool_messages) | ||
| chat_ctx.items.extend( |
There was a problem hiding this comment.
Maybe
agents/livekit-agents/livekit/agents/voice/agent.py
Lines 966 to 972 in e87d3ee
There was a problem hiding this comment.
(Just adding what we talked about in meeting for posterity)
That part works well, I think it's that the tool response is generated from the turn's local snapshot, not from the agent's chat_ctx when _pipeline_reply_task_impl copies the ctx at the start of the turn and the same local copy is passed back in for the tool reply
The copy is from before the tool execution, so the items the merge added to agent._chat_ctx aren't in there so the next turn sees them, but the immediate tool reply doesn't, so sometimes the tool reply asks for things that have actually already been given between the context that it copied and the current state.
… reply context When a function tool awaits an inline AgentTask, the sub-conversation it runs is merged into the agent's chat_ctx at handoff-return. The reply generated after tool execution used a chat_ctx snapshot taken before the tool ran, so it was blind to everything said inside the task and would re-ask for fields the task had already collected. Merge the agent's chat_ctx back into the snapshot (after adding the tool messages, so merge() dedups them) before generating the tool reply, mirroring the existing update_instructions() refresh.
ShayneP
force-pushed
the
ShayneP/agent-task-chat-ctx-refresh
branch
from
8746591 to
04ac3be
Compare
theomonnom
left a comment
theomonnom
left a comment
There was a problem hiding this comment.
lgtm! discussed in a meeting
| # Conversational analog of the update_instructions() refresh below. | ||
| # tool_messages must be added first so merge() dedups them by id. | ||
| chat_ctx.items.extend(tool_messages) | ||
| chat_ctx.merge(self._agent._chat_ctx, exclude_instructions=True) |
There was a problem hiding this comment.
🚩 exclude_instructions=True excludes all system/developer messages, not just the instruction message
The merge call at line 3056 passes exclude_instructions=True, which in chat_context.py:583-588 skips ALL items where item.type == 'message' and item.role in ['system', 'developer']. This means if the sub-conversation (run inside an inline AgentTask) produced any system or developer messages beyond instructions — e.g., system-level context injected by the sub-agent — those would be excluded from the merge into the tool-response generation context. This appears intentional since the update_instructions call at lines 3061-3065 handles instruction refresh separately, but it's worth noting that non-instruction system messages from sub-conversations will be silently dropped.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
When a function tool awaits an inline
AgentTask, the sub-conversation it runs is merged into the agent'schat_ctxat handoff-return. But the reply generated after tool execution uses achat_ctxsnapshot taken before the tool ran, so it's blind to everything said inside the task — in practice the agent re-asks for fields the task already collected.This refreshes the snapshot with any chat items added during tool execution before generating the tool reply, mirroring the existing
update_instructions()refresh just below it.Found while benchmarking the hotel receptionist example, where the card-collection
AgentTaskwould complete and the agent would immediately ask for the card number again.